HEAD
仅使用前视投影 得到的效果更好?可能是网络的规模不够大
尝试仅使用前视投影+sn
删除前视投影 仅保留pc分支:
loss非常不稳定 生成图像效果差
将pc分支的浅层特征经过down和res分支
另外将pc特征加入到discriminator中 或者从discriminator中取消投影输入
SN: spectral norm
在discriminator中删除conditional输入之后, feature matching loss和vgg loss的初始值变得非常高,这可能是因为没有conditional输入之后,discriminator在开始接受到的输入就完全是混乱的初始生成图像 没有与真实图像的feature比较类似的条件输入;但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定),后期也下降到和有conditional输入的discriminator一样的情况了
最终的测试结果没有加入conditional输入好
目前来看Generator输入的上采样图像的作用比较大
静止场景:
0016 0017 0019(步行街场景 行人多 光照复杂)
尝试noVGG 效果很差
尝试pixel shuffle 解决棋盘格效应
object sensitive loss
尝试对gamma进行随机增强,生成的图像亮度更高,在50轮的时候视觉效果更好一些,但是没有最开始的点云分支好
尝试加入pointnet2的head, 收敛速度变慢, 训练集有一些图像效果还可以,对训练集拟合效果不错
激光雷达+图像 去阴影
和相机重建出来的对比
将测试集换为tracking与object的差集 效果不错
尝试在resblock前加入pc特征 效果比resblock前加入的细节和边缘要好
对点云的intensity进行归一化
在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大 从[7,12]下降到了[1, 2], 这可能是因为SN实现了Lipschitz连续条件 但是只使用SN时拟合效果非常差
无法复现之前pix2pix的效果了 目前的改动:Gamma:无效 flip:无效 batchsize 有一些效果,现在感觉可能是数据集的问题
尝试直接用前视投影的点云提特征?
加入膨胀卷积之后效果很好
远处的弱化 从loss上考虑
LiCAM segmentation 实验
Accumulating evaluation results... │tmpfs 38G 3.8G 34G 10% /run
DONE (t=0.06s). │/dev/sda2 439G 393G 24G 95% /
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.590 │tmpfs 189G 736M 188G 1% /dev/shm
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.882 │tmpfs 5.0M 0 5.0M 0% /run/lock
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.646 │tmpfs 189G 0 189G 0% /sys/fs/cgroup
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402 │/dev/sda1 511M 3.7M 508M 1% /boot/efi
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689 │tmpfs 38G 28K 38G 1% /run/user/108
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866 │tmpfs 38G 22G 16G 58% /run/user/1000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.180 │tmpfs 38G 0 38G 0% /run/user/1001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.613 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.625 │[sudo] password for bdbc201:
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907
LiCAM 测真实图像
DONE (t=0.20s). │/dev/sda2 439G 393G 24G 95% /
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.613 │tmpfs 189G 736M 188G 1% /dev/shm
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.885 │tmpfs 5.0M 0 5.0M 0% /run/lock
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.689 │tmpfs 189G 0 189G 0% /sys/fs/cgroup
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397 │/dev/sda1 511M 3.7M 508M 1% /boot/efi
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737 │tmpfs 38G 28K 38G 1% /run/user/108
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886 │tmpfs 38G 22G 16G 58% /run/user/1000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.195 │tmpfs 38G 0 38G 0% /run/user/1001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.646 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.662 │[sudo] password for bdbc201:
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932
真实图像测SPADE
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.449
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.678
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.485
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.161
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816
真实图像测皮鞋pix2pixHD
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.146
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.301
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.130
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.092
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.168
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.168
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442
真实图像测LiCAM
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.388
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.636
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.407
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.152
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.431
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847
真实图像测SPADE-LICAM-Intensity
直接High res
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.371
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.586
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.410
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.141
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.402
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.402
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858
Low res 2 High res
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.365
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.583
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.399
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.142
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.398
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.398
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829
目前看应该是需要平滑的conditional input(这个地方可以abalation study)
真实图像测SAPDE-LICAM
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.337
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.550
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.362
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.135
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.376
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.376
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814
真实图像测pix2pix
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.007
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.010
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.010
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014
真实图像测cycle
None 没有结果
改用OMP多核实现之后 生成1M个点可以从90s提高到10s
使用光栅化过程的zbuffer解决了深度遮挡的问题 可以考虑不用光线追踪 但是仍然是串行实现的
ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞,孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓,但是在semantic label里,相近id代表完全不同的类别,因此使用闭运算,先膨胀再腐蚀,可以很好的消除孔洞
闭运算后
闭运算前
**** SPADE 没有crop图像的上面120个pixel
U-Net或者hourglass 应用在point network中
点云segmentation 之后用结果对应图像的纹理做tracking
标定 想通过射线投影模型,把方差较大的点投影会拟合的平面上 但是由于x方向方差过大 拟合的标定板投影实际上出现了仿射变换 无法构成矩形棋盘
在分割棋盘格时,最开始是找Z轴方向密度变化最大的一个点作为分割点,但是由于horiozon扫出来的点扫镖器边缘密度不均匀,很容易在棋盘中间造成分割点。为了解决这个问题,每次找到Topk个可能的分割点,计算每个点与棋盘格长度的差,找到差距最小的那个点作为真正的分割点,这样可以保证这个点是可以覆盖整个棋盘的
尝试了用求得的外参反向计算内参,但是发现求得的结果还是和内参输入是一样的,这可能是因为固定了错误的内参,反向计算的内参都是在过拟合这个外参
对比手工标注和棋盘格标到相同的RMSE所需要的标定板摆放次数
** feature map在空间上进行泊松融合
轨迹叠加 追踪
* 寻找最优的摆放位置,可以先尝试暴力搜索,在给定的一组lidar-camera pair中找到一组子集,使得准确率最高
扰动LiDAR点云的坐标系,这样就构成了不同位置的点云-图像pair
legoloam 在狭窄走廊+行人遮挡会出现degeneration; 这个degeneration还和建图开始的位置有关, 因为不同的建图开始位置会导致不同的局部地图, 从而对mapoptimization的局部地图匹配造成影响;这个问题在lio-sam中更严重
BBA 在KITTI鸟瞰图上直接检测 效果特别差
AP@0.3只有0.14512388417054484
AP@0.5只有0.02
nuscene part1
train
1400短序列
val 170
Mean pixel classification accuracy: 0.9270618313934371
mean cat accuracy of Bg: 0.9455821795090201
mean cat accuracy of Vehicle: 0.5206497586391847
mean cat accuracy of Ped: 0.44901742677048573
mean cat accuracy of Bike: 0.0
mean cat accuracy of Others: 0.588819095477387
mean instance acc: 0.5008136920792154
test 500
Mean pixel classification accuracy: 0.9226152206242199
mean cat accuracy of Bg: 0.9450818408534926
mean cat accuracy of Vehicle: 0.574332482538324
mean cat accuracy of Ped: 0.5306154084638041
mean cat accuracy of Bike: 0.0
mean cat accuracy of Others: 0.5026899037205982
mean instance acc: 0.5105439271152438
每一帧的相对时间分布是有相位差的, 比如前25%收到的点, 并不在帧数据的前25%位置, 而是在整个点云中滑动. 这个滑动可能通过运动模型计算出来.
仅使用前视投影 得到的效果更好?可能是网络的规模不够大
尝试仅使用前视投影+sn
删除前视投影 仅保留pc分支:
loss非常不稳定 生成图像效果差
将pc分支的浅层特征经过down和res分支
另外将pc特征加入到discriminator中 或者从discriminator中取消投影输入
SN: spectral norm
在discriminator中删除conditional输入之后, feature matching loss和vgg loss的初始值变得非常高,这可能是因为没有conditional输入之后,discriminator在开始接受到的输入就完全是混乱的初始生成图像 没有与真实图像的feature比较类似的条件输入;但是loss下降的趋势还比较稳定(即没有造成训练过程的不稳定),后期也下降到和有conditional输入的discriminator一样的情况了
最终的测试结果没有加入conditional输入好
目前来看Generator输入的上采样图像的作用比较大
静止场景:
0016 0017 0019(步行街场景 行人多 光照复杂)
尝试noVGG 效果很差
尝试pixel shuffle 解决棋盘格效应
object sensitive loss
尝试对gamma进行随机增强,生成的图像亮度更高,在50轮的时候视觉效果更好一些,但是没有最开始的点云分支好
尝试加入pointnet2的head, 收敛速度变慢, 训练集有一些图像效果还可以,对训练集拟合效果不错
激光雷达+图像 去阴影
和相机重建出来的对比
将测试集换为tracking与object的差集 效果不错
尝试在resblock前加入pc特征 效果比resblock前加入的细节和边缘要好
对点云的intensity进行归一化
在Discriminator中取消SN后面的BN之后 Gan的Feature matching loss的量级下降了很大 从[7,12]下降到了[1, 2], 这可能是因为SN实现了Lipschitz连续条件 但是只使用SN时拟合效果非常差
无法复现之前pix2pix的效果了 目前的改动:Gamma:无效 flip:无效 batchsize 有一些效果,现在感觉可能是数据集的问题
尝试直接用前视投影的点云提特征?
加入膨胀卷积之后效果很好
远处的弱化 从loss上考虑
LiCAM segmentation 实验
Accumulating evaluation results... │tmpfs 38G 3.8G 34G 10% /run
DONE (t=0.06s). │/dev/sda2 439G 393G 24G 95% /
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.590 │tmpfs 189G 736M 188G 1% /dev/shm
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.882 │tmpfs 5.0M 0 5.0M 0% /run/lock
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.646 │tmpfs 189G 0 189G 0% /sys/fs/cgroup
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.402 │/dev/sda1 511M 3.7M 508M 1% /boot/efi
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.689 │tmpfs 38G 28K 38G 1% /run/user/108
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.866 │tmpfs 38G 22G 16G 58% /run/user/1000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.180 │tmpfs 38G 0 38G 0% /run/user/1001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.613 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.625 │[sudo] password for bdbc201:
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.453 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.715 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907
LiCAM 测真实图像
DONE (t=0.20s). │/dev/sda2 439G 393G 24G 95% /
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.613 │tmpfs 189G 736M 188G 1% /dev/shm
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.885 │tmpfs 5.0M 0 5.0M 0% /run/lock
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.689 │tmpfs 189G 0 189G 0% /sys/fs/cgroup
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.397 │/dev/sda1 511M 3.7M 508M 1% /boot/efi
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.737 │tmpfs 38G 28K 38G 1% /run/user/108
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886 │tmpfs 38G 22G 16G 58% /run/user/1000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.195 │tmpfs 38G 0 38G 0% /run/user/1001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.646 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS sudo kill 3771
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.662 │[sudo] password for bdbc201:
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.458 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.778 │(base) ➜ sTrain-sTest_LiCAM_cas101_MS du -sh
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.932
真实图像测SPADE
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.449
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.678
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.485
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.783
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.161
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.254
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.816
真实图像测皮鞋pix2pixHD
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.146
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.301
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.130
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.038
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.179
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.407
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.092
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.168
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.168
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.043
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.209
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.442
真实图像测LiCAM
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.388
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.636
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.407
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.151
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.511
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.809
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.152
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.431
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.186
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.553
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.847
真实图像测SPADE-LICAM-Intensity
直接High res
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.371
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.586
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.410
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.111
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.502
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.813
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.141
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.402
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.402
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.137
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.858
Low res 2 High res
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.365
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.583
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.399
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.112
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.791
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.142
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.398
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.398
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.138
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.530
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829
目前看应该是需要平滑的conditional input(这个地方可以abalation study)
真实图像测SAPDE-LICAM
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.337
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.550
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.362
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.102
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.453
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.135
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.376
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.376
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.494
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.814
真实图像测pix2pix
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.007
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.010
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.010
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.015
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.003
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.014
真实图像测cycle
None 没有结果
改用OMP多核实现之后 生成1M个点可以从90s提高到10s
使用光栅化过程的zbuffer解决了深度遮挡的问题 可以考虑不用光线追踪 但是仍然是串行实现的
ray tracng 之后的结果会在属于同一个类别的大片联通区域产生孔洞,孔洞中的数值是与该区域的类别非常相似的数值。这在图像中无所谓,但是在semantic label里,相近id代表完全不同的类别,因此使用闭运算,先膨胀再腐蚀,可以很好的消除孔洞
闭运算后
闭运算前
**** SPADE 没有crop图像的上面120个pixel
U-Net或者hourglass 应用在point network中
点云segmentation 之后用结果对应图像的纹理做tracking
标定 想通过射线投影模型,把方差较大的点投影会拟合的平面上 但是由于x方向方差过大 拟合的标定板投影实际上出现了仿射变换 无法构成矩形棋盘
在分割棋盘格时,最开始是找Z轴方向密度变化最大的一个点作为分割点,但是由于horiozon扫出来的点扫镖器边缘密度不均匀,很容易在棋盘中间造成分割点。为了解决这个问题,每次找到Topk个可能的分割点,计算每个点与棋盘格长度的差,找到差距最小的那个点作为真正的分割点,这样可以保证这个点是可以覆盖整个棋盘的
尝试了用求得的外参反向计算内参,但是发现求得的结果还是和内参输入是一样的,这可能是因为固定了错误的内参,反向计算的内参都是在过拟合这个外参
对比手工标注和棋盘格标到相同的RMSE所需要的标定板摆放次数
** feature map在空间上进行泊松融合
轨迹叠加 追踪
* 寻找最优的摆放位置,可以先尝试暴力搜索,在给定的一组lidar-camera pair中找到一组子集,使得准确率最高
扰动LiDAR点云的坐标系,这样就构成了不同位置的点云-图像pair
legoloam 在狭窄走廊+行人遮挡会出现degeneration; 这个degeneration还和建图开始的位置有关, 因为不同的建图开始位置会导致不同的局部地图, 从而对mapoptimization的局部地图匹配造成影响;这个问题在lio-sam中更严重
BBA 在KITTI鸟瞰图上直接检测 效果特别差
AP@0.3只有0.14512388417054484
AP@0.5只有0.02
nuscene part1
train
1400短序列
val 170
Mean pixel classification accuracy: 0.9270618313934371
mean cat accuracy of Bg: 0.9455821795090201
mean cat accuracy of Vehicle: 0.5206497586391847
mean cat accuracy of Ped: 0.44901742677048573
mean cat accuracy of Bike: 0.0
mean cat accuracy of Others: 0.588819095477387
mean instance acc: 0.5008136920792154
test 500
Mean pixel classification accuracy: 0.9226152206242199
mean cat accuracy of Bg: 0.9450818408534926
mean cat accuracy of Vehicle: 0.574332482538324
mean cat accuracy of Ped: 0.5306154084638041
mean cat accuracy of Bike: 0.0
mean cat accuracy of Others: 0.5026899037205982
mean instance acc: 0.5105439271152438
vid2vid训练过程中,对flow分支的依赖比较高, flow收敛了之后generator loss会有大幅下降, 同时图像质量开始变高
对于horizon,投影到range image上是具有局部性的, 但是由如果忽略了叠加,会使得这个局部的range image变得稀疏